Earning for D Eep N Eural N Etworks

In industrial machine learning pipelines, data often arrive in parts. Particularly in the case of deep neural networks, it may be too expensive to train the model from scratch each time, so one would rather use a previously learned model and the new data to improve performance. However, deep neural networks are prone to getting stuck in a suboptimal solution when trained on only new data as compared to the full dataset. Our work focuses on a continuous learning setup where the task is always the same and new parts of data arrive sequentially. We apply a Bayesian approach to update the posterior approximation with each new piece of data and find this method to outperform the traditional approach in our experiments. 1 BAYESIAN INCREMENTAL LEARNING Recent work has shown promise in incremental learning; for example, a set of reinforcement learning problems have been successively solved by a single model with a help of weight consolidation (Kirkpatrick et al., 2016) or Bayesian inference (Nguyen et al., 2017). In this work we focus on a specific incremental learning setting – we consider a single fixed task when independent data portions arrive sequentially. We formulate a Bayesian method for incremental learning and use recent advances in approximate Bayesian inference (Kingma & Welling, 2013; Kingma et al., 2015; Louizos & Welling, 2017) to obtain a scalable learning algorithm. We demonstrate the performance of our method on MNIST and CIFAR-10 is improved relative to a naive fine-tuning approach and can be applied to a conventional (non-Bayesian) pre-trained DNN. Consider an i.i.d. datasetD = {xi, yi}i=1. In an incremental learning setting, this dataset is divided into T parts D = {D1, . . . ,DT }, which arrive sequentially during training. The goal is to build an efficient algorithm that takes a model, trained on the first t−1 units of dataD1, . . . ,Dt−1, and retrain it on a new unit of data Dt without access to D1, . . . ,Dt−1 and without forgetting dependencies. The most naive deep learning approach for incremental learning is to apply the Stochastic Gradient Descent (SGD) updates with the same loss function on the new data parts, to fine-tune the model. However, in that case, the model is likely to converge to a local optima on a new data unit without saving the information learned from the previous parts of the data. The Bayesian framework is a powerful tool for working with probabilistic models. It allows to estimate the posterior distribution p(w | D1, . . . ,Dt) over the weights w of the model. We can use the Bayes rule to sequentially update the posterior distribution in the incremental learning setting: p(w | D1, . . . ,Dt) ∝ p(Dt |w)p(w | D1, . . . ,Dt−1) (1) Unfortunately, in most cases the posterior distribution p(w | D1, . . . ,Dt) is intractable, so we can use stochastic variational inference (Hoffman et al., 2012) to approximate it. In the next section we present a scalable method for incremental learning, and study different variational approximations of the posterior distribution.